Skip to content

Added data_selection support to S3 SQS source#6148

Merged
dlvenable merged 1 commit into
opensearch-project:mainfrom
mzurita-amz:feature/s3-sqs-data-selection
Oct 8, 2025
Merged

Added data_selection support to S3 SQS source#6148
dlvenable merged 1 commit into
opensearch-project:mainfrom
mzurita-amz:feature/s3-sqs-data-selection

Conversation

@mzurita-amz

@mzurita-amz mzurita-amz commented Oct 8, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Enable SQS-based S3 source to specify whether to process data only, metadata only, or both data and metadata, matching the functionality available in scan-based S3 source.

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Testing

Verified the following config produces no S3 metadata

data_selection: data_only

2025-10-08T14:56:26,405 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Received 1 messages from SQS. Processing 1 messages.
2025-10-08T14:56:26,407 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.S3ObjectWorker - Read S3 object: [bucketName=dataprepper-test-bucket-1759926844, key=logs/clean/clean-test.json]
2025-10-08T14:56:27,694 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Deleted 1 messages from SQS. [ef7a587d-5c70-42e0-9802-062721cef19c]
{"message":"{\"test\": \"data_only_example\"}"}

data_selection: metadata_only

2025-10-08T15:02:01,637 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.S3ObjectWorker - Read S3 object: [bucketName=dataprepper-test-bucket-1759926844, key=logs/final-meta/final-metadata.json]
2025-10-08T15:02:02,058 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Deleted 1 messages from SQS. [66b2432b-42b8-424c-9de7-bf96f2cbc52e]
{"bucket":"dataprepper-test-bucket-1759926844","length":32,"time":1759932116.000000000,"key":"logs/metadata-clean/metadata-clean.json"}
{"bucket":"dataprepper-test-bucket-1759926844","length":27,"time":1759932049.000000000,"key":"logs/final-meta/final-metadata.json"}

data_selection: data_and_metadata

2025-10-08T15:04:49,276 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Received 1 messages from SQS. Processing 1 messages.
2025-10-08T15:04:49,279 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.S3ObjectWorker - Read S3 object: [bucketName=dataprepper-test-bucket-1759926844, key=logs/final-combined/final-combined.json]
2025-10-08T15:04:50,486 [s3-source-sqs-1] INFO  org.opensearch.dataprepper.plugins.source.s3.SqsWorker - Deleted 1 messages from SQS. [7c86a265-0a48-4e83-9265-59cbd9234830]
{"message":"{\"test\": \"data_and_metadata_final\"}","s3":{"bucket":"dataprepper-test-bucket-1759926844","key":"logs/final-combined/final-combined.json"},"parsed_log":{"test":"data_and_metadata_final"}}

dlvenable
dlvenable requested changes Oct 8, 2025
View reviewed changes

@dlvenable dlvenable left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @mzurita-amz for this contribution!

Comment thread ...rce/src/main/java/org/opensearch/dataprepper/plugins/source/s3/configuration/SqsOptions.java Outdated
@Min(1)
private Integer maxReceiveAttempts;

@JsonProperty("data_selection")

@dlvenable dlvenable Oct 8, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should move this to the root of the configuration. If it is present, then use that for either SQS or scan. We can allow scan options to override this for now.

The sqs configuration is intended to control how the SQS interactions are handled, but this is new configuration is a configuration on how we handle S3.

@mzurita-amz mzurita-amz Oct 8, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the configuration to the s3 config. Thanks!

@dlvenable dlvenable linked an issue Oct 8, 2025 that may be closed by this pull request
DataPrepper S3-SQS source config should support data_selection similar to S3 Scan configs #6149
Closed
@mzurita-amz mzurita-amz force-pushed the feature/s3-sqs-data-selection branch 3 times, most recently from eaa6345 to ccc4e9e Compare October 8, 2025 15:58
@mzurita-amz
Added data_selection support to S3 SQS source
83ace93 
Signed-off-by: Manuel Mangas Zurita <mzurita@amazon.com>
@mzurita-amz mzurita-amz force-pushed the feature/s3-sqs-data-selection branch from 037ac0a to 83ace93 Compare October 8, 2025 16:51
@mzurita-amz mzurita-amz marked this pull request as ready for review October 8, 2025 17:11
@mzurita-amz mzurita-amz requested a review from dlvenable October 8, 2025 17:11
dlvenable
dlvenable approved these changes Oct 8, 2025
View reviewed changes

@dlvenable dlvenable left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @mzurita-amz for this great contribution!

@dlvenable dlvenable merged commit d252490 into opensearch-project:main Oct 8, 2025
45 of 47 checks passed
@mzurita-amz mzurita-amz deleted the feature/s3-sqs-data-selection branch November 3, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dlvenable dlvenable dlvenable approved these changes
@kkondaka kkondaka kkondaka approved these changes
@sb2k16 sb2k16 Awaiting requested review from sb2k16 sb2k16 is a code owner
@chenqi0805 chenqi0805 Awaiting requested review from chenqi0805
@engechas engechas Awaiting requested review from engechas
@san81 san81 Awaiting requested review from san81 san81 is a code owner
@srikanthjg srikanthjg Awaiting requested review from srikanthjg srikanthjg is a code owner
@graytaylor0 graytaylor0 Awaiting requested review from graytaylor0 graytaylor0 is a code owner
@dinujoh dinujoh Awaiting requested review from dinujoh dinujoh is a code owner
@KarstenSchnitter KarstenSchnitter Awaiting requested review from KarstenSchnitter KarstenSchnitter is a code owner
@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DataPrepper S3-SQS source config should support data_selection similar to S3 Scan configs

3 participants

@mzurita-amz @dlvenable @kkondaka